Content addressable information encapsulation, representation, and transfer

ABSTRACT

Representing a number of assets on an originating computer begins with selecting the assets to be represented. Cryptographic hash asset identifiers are generated; each of the asset identifiers is computed using the contents of a particular asset. The asset identifier is a content-based or content-addressable asset name for the asset and is location independent. An asset list is generated that includes the asset identifiers computed from the assets. A cryptographic hash asset list identifier is generated that is computed from the asset list. The asset list identifier is stored for later retrieval. The assets selected are also stored for safekeeping either locally or on a computer network. In the event of loss of the files from the originating computer, the asset list identifier is retrieved. Using the asset list identifier, the original asset list is found and retrieved from its safe location. The asset identifiers from the retrieved asset list are used to find and retrieve the individual assets from their backup locations. The assets are verified by recomputing the cryptographic hash asset identifier for each asset retrieved and comparing it to the asset identifier from the asset list. The MD5 algorithm is used for the cryptographic hash function. Assets are retrieved using a multicast protocol. A series of importer programs searches for assets to retrieve in progressively more remote locations. Assets are retrieved whole or in segments.

This application claims priority of U.S. provisional patent applicationNo. 60/072,316, filed Jan. 23, 1998, which is incorporated herein byreference for all purposes. This application is related to co-pendingU.S. patent application Ser. No. ______ (Attorney Docket No. WRSHP003),filed concurrently herewith, which is incorporated herein by referencefor all purposes.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to methods and apparatuses forencapsulating information, identifying the information, representing theinformation, and facilitating the transfer of the information betweenusers, between remote storage and an originating user, or between remotestorages using computers and digital telecommunication networks.

2. Description of the Related Art

Digital information must often be identified to be in a particularstate, denoted by the status of an asset (such as a file) as of someevent or time. Such assets include traditional data files, multimediafiles and fragments, records from structured databases, or any otherstring of digital information used wholly or in part by some applicationor device. Digital information is highly subject to change and fewmethods are available to inspect the contents of the digital informationto reliably recognize whether it has been changed since some prior timeor event. Normal attempts to improve or perfect the content, inadvertentcommands or actions which change the content, or tampering by othersunknown to primary owners of the digital information are difficult todetect. As such, computers users have no convenient mechanism forestablishing the origin or integrity of particular content versions.

Another problematic attribute of digital information (such as a computerfile) is that copies may exist which are identical in content but differin the meta data that the computer system uses to relate to the digitalinformation. Such meta data includes the date/time recorded for thecreation or last modification of the file, the file name associated withthe file and other information. The meta data may imply that otherwiseidentical copies of digital information are different when in fact theyare not. Such confusion makes it difficult to avoid unnecessaryduplication of content on a single computer or on a collection ofcomputers on a network. This confusion may also result in theunnecessary coping of such data files across networks or from othermedia when, in fact, a particular data file needed is readily availableon a computer system or network already.

The existence of a particular file under multiple names has acounterpart problem. Data on computer systems can generally only beaccessed through identifiers or location mechanisms which to a greateror lesser extent include information about the location of the file inthe storage of the computer. That means that a user accesses the datathrough stored or remembered names which include elements which arereadily changed by others. For example, files within a sub-directory areat risk if someone changes the sub-directory name. If changed, the pathto a file becomes invalid, and all of the stored or remembered names offiles become invalid as well. This fragile approach to location of databy location leads to many kinds of problems for users and administratorsof computer systems, particularly those working with networked systems.

Finally, there is no convenient way for computer users to identifycollections of specific versions of digital files. No robust mechanismexists for computers or their users to refer to collections of specificcopies or versions of digital files without creating a new entity whichincorporates copies of the files into a new form. Many mechanisms havebeen created to combine such copies into what are commonly calledarchive files. Examples of archive utilities include the “tar” archivingfacility common on UNIX systems and the various “zip” programs onpersonal computers. Such solutions create additional copies which areoften proliferated to many systems. The difficulty of such solutions isthat often exact digital copies of many of the files in an archive arealready present on the systems to which they are copied. In fact, onmany computer systems there are many copies of digital files whosecontents are exactly the same. This duplication of identical content isdifficult to avoid using existing techniques.

The result of these problems is that duplicate copies of digital filesare frequently stored on computer storage devices (at expense to theowner of the system) or transferred on media or telecommunicationsdevices (at further expense to the system owner and thetelecommunications provider). This duplication strains limited resourcesand causes needless confusion on local private networks (local areanetworks, for example) and on collections of systems connected bydigital telecommunication networks. One problem with extra copies isthat one might think they are different when they are in fact the same(and copies are needlessly stored), or when they are different, onemight think they are the same because of the same file name.

The inability of systems to reliably distinguish different versions offiles with the same identifier or to recognize identical files withdifferent identifiers wastes network resources and creates confusionwhen files are transferred between users of a network. Often, it isessential that users know that they are working on the same document orknow that they are working with the same version of an application. Forexample, when an electronic mail (e-mail) message is sent from one userto another, an attached computer file containing an application or adocument is often sent as well. Files may also need to be transferred sothat applications can be distributed. Sending an e-mail message with anattached file or using a point-to-point scheme in a network todistribute files can be inefficient in terms of the amount of networkbandwidth that is used. For example, when a user attaches a number offiles to an e-mail message, it may be that a copy of one or more ofthose files is already stored on the intended recipient's hard drive. Insuch a case, the network bandwidth used to transfer the attached filesis wasted. If the files could be reliably identified and the files'contents could be reliably verified then the recipient could simplyretrieve the files from his own hard drive or from a local networkserver and verify that they are indeed the files that are attached tothe e-mail message.

A similar problem occurs in managing computers on a network and makingsure that the computers are configured in a certain way with certainapplications. For example, when a small change is made to an operatingsystem or to hardware that is available to the network, certain filesmay need to be transferred to each computer on the network. A givencomputer may have most or almost all of the necessary files loaded andonly a few files may need to be provided or updated from a centralsource. In many cases, the requesting computer and the source computerare far from one another and are connected by a data link that operatesat a slower speed than a local data link would operate. Currently it isnecessary to keep track of both the files that are on the requestingcomputer and the files that need to be added so that proper updates canbe made. It would be useful if there existed a way to specify all of thefiles that are to be transferred and to encapsulate that specificationin such a way that would allow the files to be retrieved from the mostconvenient place (locally, if possible). It would further be useful ifsuch a method would allow the files to be reliably verified as thecorrect files.

When files are distributed on a local area network (LAN) from a sourceoutside the LAN, the problem can be even more serious. For example, whena company such as Netscape Communications Corporation provides a new webbrowser on their web site, hundreds or even thousands of employees at asingle company attempt to download the browser from Netscape's web site.This is perhaps the most inefficient way for the required software to bedistributed within a company. It would be more efficient, for example,if one coworker could reliably retrieve needed files from another. Ifthe necessary files could be somehow uniquely identified in a mannerthat would allow the actual data in the files to be obtained from themost convenient source, then all of the outside bandwidth used up whenall the users download files from an outside source could be saved. Inaddition, users would obtain access to the files much faster as well.

The problem of specifying a set of files to be stored on variouscomputers and ensuring that the correct files are stored on thecomputers in a network is described in U.S. Pat. No. 5,581,764 issued toFitzgerald et al. Fitzgerald teaches a method of distributing resourcesover a computer network. The method involves generating Already Have andShould Have lists for each of the computers on the network and comparinga Last Updated Date/Time (LUDT) field in the Should Have list to a LastSynchronized Date/Time (LSDT) in the Already Have list. The differencesbetween Should Have lists and Already Have lists for individualcomputers are used to determine which items must be compared to updateindividual desktops. This mechanism is dependent on the integrity ofsystem clocks and date settings which are unreliable due to accidentalor malicious entry of false settings. Furthermore, the mechanism failsin principle when dealing with the identification of identical filesfrom different systems. An alternative to the Fitzgerald method thatwould not require detailed comparisons of update and synchronizationtimes yet would still allow files to be reliably specified and wouldallow needed files to be reliably identified would be useful.

U.S. Pat. No. 5,710,922 issued to Alley et al. describes a method forsynchronizing and archiving information between computer systems. Therecords are identified with a unique identification indicia and anindicia that indicates the last time that the record was altered. Usingthe time of the last synchronization information, each of the selectedrecords that was added to or deleted from one of the computer systemssince the last synchronization is identified and added to or deletedfrom the computer system. Certain techniques and operations can falselyindicate changes to records which have not, in fact, changed.Furthermore, identical copies of digital files on different systems arenot readily recognized as the same because the mechanism in Alleyprovides no mechanism to do so. Again, it would be useful if a methodfor synchronizing file systems could be developed that would not requireor depend upon analysis of update and synchronization times.

In general, there is a need for a more reliable, flexible and verifiableway of specifying states of known data assets (such as computer files)and of providing access to those unique data assets, particularly overnetworks. Currently, network sites that are sources of data may bemirrored and various load-balancing schemes have been devised fordistributing load among servers that provide data. However, no trulydistributed system has been devised for sharing and providing access todata whereby data may be reliably and automatically retrieved from anyplace where it may be found on a network, instead of from specifiedlocations which are designed to store and provide access to data.

In view of the foregoing, there is a need for methods and apparatusesthat reliably and verifiably transfer files while allowing the site thatis receiving the files to obtain the files from the most convenientsource. Further, it is desirable for such techniques to obtain files inan efficient manner, to obtain the files locally if possible, and toverify that the content of an obtained file is the same as the contentof the file that is intended to be transferred. There is also a need formethods and apparatuses that minimize the data stored or transferredwithin a system or network. It would be desirable for such techniques toprovide a reliable mechanism for identifying, locating, and accessingdata by its contents rather than by exclusively using the meta datatraditionally stored on computer systems.

SUMMARY OF THE INVENTION

Accordingly, a system and method are disclosed for representing digitalinformation in an electronic paper clip, or “e-CLIP” (™). An e-CLIP is areproducible, reliably unique identifier for a collection of digitalinformation, derived from the content of the digital information itself.In one embodiment, an e-CLIP is an alphanumeric reference. An e-CLIP mayrepresent a file, a group of files, a group of file identifiers, orother collections of data or database information. Such othercollections of data or database information might include selecteddatabase records from a relational, hierarchic, network or other formatdatabase, selected frames or clips from digital audio or video streams,messages from streams of message records or files, log entries fromaudit or status logs of systems, subsystems and/or applications or anyother digital assets, the status of which at some instance in time is ofunique importance in some context. The original form or context of eachdigital asset is irrelevant so long as applications provide each suchasset uniquely to the mechanism or system embodying this invention. Itoperates on each unique asset and associated meta-data as described toproduce a unique and useful identifier which enables creation ofpersistent storage of the related assets for future reproduction of theoriginals.

A cryptographic hash function is used to compute an identifier for thedata being represented. Each binary asset is treated as a potentiallyunique binary sequence. That is to say that any binary entity has aseries of binary digits which, in sequence, follow a potentially uniquepattern of finite length. Thus, a binary asset at an instant in time isa binary sequence which may or may not be unique. The use of acryptographic hash function establishes a digital fingerprint orsignature that virtually uniquely identifies the binary sequence. Thecryptographic hash binary sequence identifier is also referred to as acontent-addressable or content based name for the data. When a group offiles or other digital assets is represented, an identifier is generatedfor each of the files using a cryptographic hash function and placed ina descriptor file. The descriptor file also includes meta data such asarbitrary directory structure (including relational or hierarchicalrelationships) information as well as file, record, or other asset metadata such as file, record, or asset name, size, date and time stamps andother descriptive data or attributes. In addition, the descriptor fileincludes context information about the creation of the collection (timeand date of creation, user ID of the creating user, etc.). Acryptographic hash descriptor file identifier (or descriptor file hash)is then computed for the descriptor file.

An e-CLIP includes the descriptor file hash and may also include a filelocator such as a file name or URL that gives a source where thedescriptor file may be obtained if it is not found locally in aconvenient storage location. The binary sequence hashes and descriptorfile hashes (a special case of binary sequence hash) are provably uniqueidentifiers of the relevant binary sequences. As such, they form afoundation for the storage and retrieval of those sequences as files,database records, or other digital entities using the hashes as assetidentifiers (keys, locators or other mechanism). Such an approach can besaid to provide “content addressable” storage as the hash is derivedfrom the binary sequence itself, the digital content.

In one aspect, the present invention is advantageous in that meta dataassociated with each file/record/asset, other data associated with eachor all of those assets, and context data about the collection is alsoincluded in the descriptor file. Thus, when the descriptor file isobtained, the recipient also receives important meta data about eachasset and context information about the collection. The meta data may beused to further verify a file/record/asset, to indicate ownership, toshow modification dates, or to provide other needed information abouteach file. In addition, an embodiment of the present invention isadvantageous when file directory structure is also included with thefile list. Having such file directory structure is helpful indetermining how to organize files amongst their respective folders. Forexample, after data is lost on a particular computer, the file list canbe used to not only identify lost files, but also to reorganize thefiles into the appropriate directory structure. Similarly, meta dataabout database records cataloged in a descriptor file can be used toidentify tables or files to which those records pertain. Such parallelscan be drawn with other forms of digital asset like audio or videoclips, etc.

An e-CLIP functions as a “key to a box of keys” where the box of keys isthe descriptor file and the keys are the binary sequence identifiers (orbinary sequence hashes). The use of the term “key” has no cryptographicor other meaning in the context of this invention. An e-CLIP is usefulfor identifying groups of files that have been backed up, are beingtransferred, etc. At some point, a user (or system) may wish to accessthe files starting with nothing more than the e-CLIP.

Once the descriptor file (or box of keys) is found using the e-CLIP,each of the files/records/assets corresponding to the binary sequenceidentifiers in the descriptor file may likewise be found using theirrespective unique binary sequence identifiers. When a particular binarysequence is obtained from a source, the cryptographic hash function isused to recompute the binary sequence identifier to verify that theasset obtained is the correct asset that was intended to be accessed.There is no restriction on the data, meta data or file system structurethat can be stored and referenced by an e-CLIP.

It should be appreciated that the present invention can be implementedin numerous ways, including as a process, an apparatus, a system, adevice, a method, or using a computer readable medium. Several inventiveembodiments of the present invention are described below.

In one embodiment, a system and method are disclosed for representing aplurality of assets (files, records, or other digital assets). Themethod includes selecting the plurality of assets (binary sequences) tobe transferred. A plurality of cryptographic hash binary sequenceidentifiers are generated for the plurality of assets. Each of theplurality of cryptographic hash asset identifiers is computed from thecontents of a particular asset A descriptor file is generated thatincludes the plurality of cryptographic hash binary sequence identifierscomputed from the plurality of assets to be transferred. A cryptographichash descriptor file identifier is generated that is computed from thedescriptor file. The computed cryptographic hash descriptor fileidentifier may be included in another list of identifiers, and so on, sothat complex structures can be reduced and represented in extremelycompact form.

In another embodiment, a method of identifying an asset is disclosed.The method includes selecting an asset to be identified. A cryptographichash asset identifier is obtained for the selected asset. A copy of theasset is obtained and the integrity of the copy of the asset is verifiedby regenerating the cryptographic hash file identifier from the copy ofthe asset and comparing to the cryptographic hash asset identifier ofthe asset being identified.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be readily understood by the followingdetailed description in conjunction with the accompanying drawings inwhich:

FIG. 1 is a flowchart illustrating a process for creating acryptographic hash descriptor file identifier of a descriptor file,including file meta data for the assets in the list.

FIG. 2 is a diagram illustrating the structure of a descriptor file.

FIG. 3 is a flowchart illustrating a process for using an e-CLIP to findboth a descriptor file and the assets specified in the descriptor file.

FIG. 4 is a flowchart illustrating a process running on an importer usedto receive requested assets and to verify binary sequence identifiers asis specified in step 316 of FIG. 3.

FIG. 5 is a flowchart illustrating one embodiment of step 402 showinghow the importer checks multicast transmissions to reconstitute assetsthat are received in portions.

FIG. 6A is a block diagram illustrating the structure of an assetrequest generated by an importer as described above in step 402.

FIG. 6B is a block diagram illustrating the structure of a data packetthat delivers file data to a requester in response to a asset request.

FIG. 7 is a block diagram illustrating one such chained set ofimporters.

FIGS. 8 and 9 illustrate a computer system 900 suitable for implementingembodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the preferred embodiment of theinvention. An example of the preferred embodiment is illustrated in theaccompanying drawings. While the invention will be described inconjunction with the preferred embodiment, it will be understood that itis not intended to limit the invention to one preferred embodiment. Onthe contrary, it is intended to cover alternatives, modifications, andequivalents as may be included within the spirit and scope of theinvention as defined by the appended claims. For example, for ease ofunderstanding, many of the figures illustrate use of the invention withtraditional computer files. As described herein, however, the presentinvention is suitable for use with any digital asset or binary sequence.

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Thepresent invention may be practiced without some or all of these specificdetails. In other instances, well known process operations have not beendescribed in detail in order not to unnecessarily obscure the presentinvention.

Overview

The present invention provides a technique and mechanism by which areliably unique binary sequence identifier (also referred to as a binarysequence hash or a cryptographic hash binary sequence identifier) isgenerated for each binary sequence in a user-defined collection ofbinary sequences (digital assets). These binary sequence identifiers arestored within a descriptor file of the present invention so that trueand accurate copies of those collected files can be identified and/orverified when a collection is reconstructed or validated. Further, areliably unique descriptor file identifier (or descriptor file hash) isgenerated for the descriptor file to serve as a representation of thecollection of files.

The present invention makes it possible to inspect any collection ofdigital assets to establish whether each asset in the collection is oris not present on a particular computer system or network without havingto provide a reference copy of the entire asset or relying onpotentially misleading extrinsic naming or locational information. Inthis way, only those assets that can be proven to be missing from thesystem or network need to be obtained from other sources in order toreproduce the collection of assets. The preferred embodiment of thepresent invention primarily deals with digital assets which are datafiles. Appropriate interfaces would make it simple to extend thepreferred embodiment of the invention to work equally well with digitalassets which were records from structured files and databases of alltypes, selections or clips from streams of multimedia data (digitalaudio or digital video, for example) or selections or subsets of otherstructured or unstructured digital data (binary sequences).

A preferred embodiment of the present invention uses one of a class ofcryptographic hash functions that uses the contents of a digital assetto produce a unique binary number by mathematical and/or logicaloperations. Such functions are commonly used in encryption of digitalinformation and an extensive body of art exists documenting alternativemethods for generating such a unique binary number for virtually anyspecific combination of digital data. A preferred embodiment uses thewell-known MD5 algorithm. It should be recognized that numerous otheralgorithms for generating reliably unique asset identifiers may be usedas well. Preferably, an algorithm should consistently produce the samebinary number for any specific instance of a digital file and such abinary number should be practically proven to be unique with areasonably high probability for the class of binary files beingidentified. With such an algorithm it could be proven that cryptographichashes over two binary sequences that result in the same cryptographichash (binary number) prove that the two binary sequences are the same.Conversely, cryptographic hashes over two binary sequences that resultin different cryptographic hashes (binary numbers) prove that the binarysequences are different. Such an algorithm simplifies the identificationof copies of a particular binary sequence.

A user-defined collection of digital assets, related meta data andcontext information are grouped to produce a descriptor file. Oneexample of a descriptor file is shown in FIG. 2. Hence, the descriptorfile can be characterized as a box or list of keys to digital assets; inaddition, it contains other information about those assets. This box ofkeys is then treated as an independent digital asset, and its own key isthen derived from its unique content. The resulting key is the “key tothe box of keys” and may be used to form an e-CLIP that represents thecollection of digital assets.

A user or system can obtain an e-CLIP from any trusted source. Thee-CLIP can then be used to find or identify a precise copy of thedescriptor file that in turn further includes the collection of assetinformation. Once a copy of an original asset is found using the presentinvention, that asset can safely be treated as a precise copy of theoriginal asset. If the asset is a descriptor file, it can be read oropened and the cryptographic hash binary sequence identifiers for thecollection of digital assets can be obtained. The files corresponding tothose binary sequence identifiers may be obtained and verified by acomparison of the provided binary sequence identifiers with binarysequence identifiers newly derived using the cryptographic hashfunction.

If the files identified in a descriptor file cannot be found, then thecollection of files cannot be reconstructed. This is a potentiallyfrustrating fact. Nevertheless, the described method provides amechanism by which collections of files can be reproduced reliably orcan be proven to be unavailable with equal reliability.

In a preferred embodiment, a descriptor file is created by generating acryptographic hash binary sequence identifier for each digital asset ina selected collection of digital assets. The cryptographic hash binarysequence identifier is generated by using a cryptographic hash functionon the actual data content of each of the assets. In some embodimentsthe entire asset is used to generate the cryptographic hash binarysequence identifier, and in other embodiments, a portion of the asset isis used. Preferably, a sufficiently large portion is used to ensure aprobability that the cryptographic hash binary sequence identifier isunique. In different embodiments, different cryptographic hash functionsare used. In a preferred embodiment, the MD5 algorithm is used togenerate a 128-bit number that represents the file. The 128-bit numberis represented as a 26-character alphanumeric string by translation tobase 36 numbers that are then mapped to the set of alphabetic andnumeric characters in the base ASCII character set. In the preferredembodiment, a flag character is included at a predetermined positionwithin the resulting string bringing the total length of the string to27 characters. This mapping is referred to as “ASCII Armoring” and iscommonly used to render binary information in a limited character setfor transmission over protocols that require content to be constrainedto alphanumeric binary coding.

As is described below, this particular representation of a cryptographichash binary sequence identifier has the advantage of being humanreadable and easily communicated for use, e.g., by being written down,transmitted by software, retrieved by data query coded into softwareapplication file requests, referenced by a content or asset managementsystem, requested in an object browser, electronically copied and pastedfrom one document to another, sent via electronic mail, etc.

A cryptographic hash function such as the MD5 algorithm is used in oneembodiment to generate the cryptographic hash binary sequence identifierbecause cryptographic hash functions have been mathematically proven tominimize the probability that similar assets will be mapped onto thesame cryptographic hash binary sequence identifier. This is importantbecause the cryptographic hash binary sequence identifier is used as aunique assets identifier and the generation of the same cryptographichash binary sequence identifier from two assets is assumed toconclusively show that the assets are identical. Conversely, it isequally useful to note binary sequences which are not the same willproduce different binary sequence identifiers and such results canconclusively show two binary sequences are not identical. The MD5algorithm produces a high confidence level and is thus highly reliableas a technique for producing a unique assets identifier.

Other hash functions br other types of functions based on the binarysequence (content) may be used to generate assets identifiers so long asthe probability of generating identical identifiers from different filesis below a threshold that is defined as acceptable.

Once the cryptographic hash file identifier for each of the selectedassets is generated, it is included in a descriptor file along withother assets information in a descriptor file. The other assetsinformation included with the file list may include directoryinformation about how the assets are organized within a computer system,as well as file names, file sizes, time and date stamps for each assets,ownership of the asset, and other asset meta data as is described below.The descriptor file may also include data about the context orimplications of the collection of assets, the purposes for which thecollection is being created, or any other information. Then, in apreferred embodiment, the descriptor file is stored in a digital file ina suitable form for the type of computer system or other environment inwhich the descriptor file resides. In other embodiments, the descriptorfile might be stored in a database or other digital repository providingconvenient, efficient, and secure storage and retrieval capabilities. Acryptographic hash binary sequence identifier for the stored descriptorfile is then computed which, in one embodiment, becomes the e-CLIP bywhich the collected assets may be referenced, found, and verified. Itshould be recognized that the e-CLIP that identifies the collection maybe produced by the same algorithm used to compute the cryptographic hashbinary sequence identifiers for the individual assets named and listedwithin the descriptor file. In other embodiments, the binary sequenceidentifier for the descriptor file is combined with other information(such as a file locator) to form the e-CLIP

e-CLIP Generation

FIG. 1 is a flowchart illustrating a process for creating acryptographic hash binary sequence identifier of a list of assets,including meta data and context data for the assets in the list. In step102, a list of assets which are to be represented is selected and theasset data, meta data, and/or context data is collected. The list ofassets may include multiple assets, only one asset, or no asset. Thelist of assets may even include previously created descriptor files orassets that include an e-CLIP. If a descriptor file contains no digitalassets (files, database records, multimedia clips, etc.), then thedescriptor file to be created may contain other data that is used tolocate and obtain digital assets using a selected scheme or may containvaluable collections of meta data and context data without reference toindependent binary sequences. In such a case, a cryptographic hashbinary sequence identifier for the descriptor file still ensures theintegrity of the data in the descriptor file.

When at least one asset is selected, in step 104 a cryptographic hashbinary sequence identifier is generated for each of the assets selected.As noted above, in one embodiment, the MD5 algorithm is used to generatethe cryptographic hash binary sequence identifier. Thus, a cryptographicbinary sequence hash is used as an asset identifier for each of theassets. In step 106, a descriptor file is created using the meta dataassociated with each asset, meta data about the assets, and context dataabout the collection, and the cryptographic hash binary sequenceidentifiers generated in step 104. An example of a descriptor file isshown in FIG. 2 below.

In step 108, a cryptographic hash is generated of the descriptor fileitself. Each of the cryptographic hash binary sequence identifiers inthe descriptor file may be thought of as a key to the digital assetwhich the cryptographic hash file identifier identifies. Thus, thedescriptor file can be thought of as a collection or “box” of keys. Thecryptographic hash binary sequence identifier of the descriptor file isreferred to as a cryptographic hash binary sequence list identifier andcan be thought of as the key to the box of keys that are listed in thedescriptor file. The cryptographic hash binary sequence list identifieris used to locate and verify the descriptor file. The contents of thedescriptor file are then in turn used to locate and verify each of theassets represented in the descriptor file. In step 110, thecryptographic hash binary sequence list identifier is converted to ASCIIformat. As noted above, in one embodiment, a 128-bit file listidentifier is converted to a 27-character base 36 ASCII string. The27-character string is thus in human readable text form and may becopied manually or electronically for processing, reference or storage.

In one embodiment, the cryptographic hash binary sequence listidentifier is stored as a bar code. This is particularly useful whenidentifying information about an object is placed on the object itself.For example, an appliance such as a microwave or a VCR could have a barcode placed on the appliance that represents a cryptographic hash binarysequence list identifier that was generated from assets that include themanual or other documentation related to the appliance. Thus, the manualand related documentation can be obtained by: scanning the bar code;reading the descriptor file identifier; obtaining the descriptor filethat corresponds to the descriptor file identifier; reading thedescriptor file and the individual binary sequence identifiers withinit; obtaining the digital assets that correspond to the binary sequenceidentifiers; and finally, reading the obtained assets that contain themanual and related documentation. Similar coding in other indices orsoftware applications can be used to specify, search for, and acquireother digital assets containing data or software code.

In step 112, the ASCII string is stored as a unique identifier ore-CLIP. The unique identifier is easily read or copied by either humanor electronic means. Next, in step 114, in certain embodiments, theunique identifier can be combined with a file locator (as an e-CLIPhint) to form the e-CLIP. The file locator indicates a possible locationof the purported descriptor file and associated digital assets (binarysequences). It should be noted that in many embodiments, e-CLIPs do notneed to include a descriptor file locator (e-CLIP hint). However, theinclusion of a descriptor file locator as the place where the descriptorfile may be found is beneficial in many instances, and especially if thedescriptor file or one or more associated digital assets is not found ina convenient location first.

Thus, the e-CLIP is represented by a unique identifier which, in oneembodiment, is a human readable version of a cryptographic hash binarysequence list identifier. The cryptographic hash binary sequence listidentifier is a unique reference to information of arbitrary size, type,complexity, and file structure. That is, the cryptographic hash binarysequence list identifier may represent any number of digital assetsfurther described by any amount of relevant meta data about file systemstructures, database relationships, multimedia content information, orother useful information. An example of a directory structure specifiedin a descriptor file is shown in FIG. 2.

FIG. 2 is a diagram illustrating the structure of a descriptor file 200.The particular descriptor file shown uses a “hyperfile” modelinglanguage (HFML) based on XML to describe the structure of thedirectories containing files as well as the files themselves. An HFML isdescribed in the provisional patent application referenced above. Ingeneral, it should be noted that implementation of an e-CLIP is notrestricted to a descriptor file written in this syntax. The HFML in thepreferred embodiment is used because it is readily parsed and can beused to generate a tree-structured directory of the files and keyscontained in the descriptor file. This example restricts itself to adescription of files and keys from a particular form of computer andsoftware. The invention provides for extension of the languages or codesused to create descriptor files to describe virtually any digital asset,relationships, and other meta and context data without limitation.

The first item in descriptor file 200 is a folder 202. A folder name 204as well as a time stamp 206 are included in folder 202. Folder 202matches up with an end folder tag 208 that marks the end of folder 202.Nested inside of folder 202 is a first nested folder 212. Folder 212includes a folder name 214 and a time stamp 216. A file 222 is includedinside of folder 212. File 222 includes a file name 224, a time and datestamp 226, a size 228, and a cryptographic hash file identifier 230generated by the MD5 algorithm and represented as a 27-character string.Likewise, folder 212 also includes a file 232. File 232 includes a filename 234, a time and date stamp 236, a size 238, and a cryptographichash file identifier 240. Folder 212 matches with an end folder tag 219.

It should be evident that an arbitrary number of folders can thus berepresented and nested within other folders as desired, so that anarbitrary tree-shaped directory can be specified with an arbitrarynumber of files specified in each of the folders of the directory. Eachof the files may include a file name and other meta data as desired plusa cryptographic hash binary sequence identifier that uniquely identifiesthe file based on the content of the file. In some embodiments, thecryptographic hash binary sequence identifier is the only identifier forthe file; in other embodiments a conventional file name is alsoassociated with the file.

Thus, it will be appreciated that when the descriptor file specified byan e-CLIP is recovered (for example, after a computer crash) and wherethe descriptor file contains definitions of computer files (possibly inaddition to other digital assets), complete file name and directoryinformation for the files that are specified by the e-CLIP is obtained.The process of retrieving the descriptor file and finding the filesspecified in the descriptor file is described in FIG. 3 below.

File Retrieval

FIG. 3 is a flowchart illustrating a process for using an e-CLIP to findboth a descriptor file and the digital assets (binary sequences) which,in this example, are files specified in the descriptor file and forputting the files in the directory structure specified by the descriptorfile. In step 302 an e-CLIP is received. The e-CLIP may be receivedembedded in an e-mail message where the e-CLIP is being used by a userto specify a set of files. Alternatively, the e-CLIP may be generatedautomatically by a network device performing the backup of the files anddirectories specified in the e-CLIP. The e-CLIP may be produced by abusiness applications, sealing the relevant digital assets relating to aparticular transaction. In addition, e-CLIPs may be generated for otherreasons by any user, network node, application or hardware device thatneeds to uniquely specify a file or group of files for some purpose.Such e-CLIPs may be embedded in and readily accessed from databaseapplications, legacy applications running on mainframes, text retrievalapplications, web sites, etc.

In step 304 the recipient of the e-CLIP broadcasts a request for thefile corresponding to the unique identifier found in the e-CLIP. Next,in step 306 the e-CLIP recipient receives a descriptor file purportingto correspond to the unique identifier. Next, in step 308 the recipientcalculates the cryptographic hash of the descriptor file received usingthe same cryptographic hash function that was used to generate theunique identifier found in the e-CLIP. In step 310 the recipientverifies that the unique identifier found in the e-CLIP matches theresult of the cryptographic hash of the descriptor file. If the uniqueidentifier is not properly verified, then control is transferred back tostep 304 and the request for the file identified in the e-CLIP is sentagain. An error message or other notification may be generated as well.

If the unique identifier is verified in step 310, then control istransferred to step 312 and the recipient builds the directory structurespecified in the descriptor file. Programming logic is applied toperform system configurations and file operations to create the requireddirectories, using programming operations such as those described byHFML, for example. Preferably, each folder in the directory is createdaccording to the specified structure. Next, in step 314 the recipient ofthe e-CLIP broadcasts a request for the files listed in the descriptorfile. FIG. 6A shows an example structure for a file request.

Responses offering copies of the requested files are analyzed and copiesof the files are retrieved from the most effective sources availableincluding local file systems, local networked file systems available tothe system on which the recipient is executing, standard networkingprotocols such as the File Transfer Protocol (FTP), or through any othernetworked protocol as may be devised or specified.

In step 316 the recipient of the e-CLIP receives the files requested andverifies the file contents by generating cryptographic hashes of thefile data and comparing the results to the file identifiers listed inthe descriptor file. If any files fail the verification test, then thosefiles are requested again and an appropriate notification is generated.The process then ends.

Thus, a recipient of an e-CLIP broadcasts a request for the descriptorfile identified by the unique identifier in the e-CLIP. Once thedescriptor file is received, the e-CLIP recipient is able to verify thatthe correct descriptor file has been recovered and then broadcastsrequests for the files specified in the descriptor file. Those files areinserted into the directory structure specified in the descriptor fileonce they are received and verified. The process for broadcastingrequests for files, receiving and verifying files, and modifying thebroadcast request is accomplished in one embodiment using an importer,which is a small program encoded preferably in the JAVA programminglanguage, or in any other suitable language.

FIG. 4 is a flowchart illustrating a process running on an importer usedto receive requested digital assets (binary sequences) which may befiles and to verify their file identifiers as is specified in step 316of FIG. 3. It should be noted that other processes and languages torequest and verify such file identifiers may be used within the spiritand scope of the invention. In step 402 the importer waits to receivefiles. When a file is received, control is transferred to step 404. Theprocess of receiving a file in parts and assembling those parts isfurther described in FIG. 5. In step 404, the importer verifies that thecryptographic hash of the file received matches the file identifier thatwas sent out requesting the file. If the file identifier is notverified, then control is transferred to step 406 where an error handleris activated. Then, in step 408 a request for the entire file isgenerated and control is transferred back to step 402.

If, however, the file is verified, then the file request list forbroadcasts is updated in step 410 and control is transferred to step412. In step 412 it is determined whether all the files have beenreceived that were specified in the descriptor file identified in thee-CLIP. If all files have been received, then control is transferred tostep 414 and it is indicated that all of the e-CLIP files have beenobtained. The process then ends. As long as all of the files have notbeen received, control transfers from step 412 back to step 402 so thatthe rest of the files may be received and checked.

It must be noted that the examples used in this description all makereference to files. The assets, however, may be references to databaserecords, video clips taken from within larger video streams, or otherdigital assets stored to be passed to other software programs orprocesses. Rather than instantiating directories and creating files withthe contents of the digital assets, the recipient would make themavailable via some other standard application programming interface. Theprocess of requesting the assets, verifying their contents, keeping themin a temporary, quarantine, holding area, and making the collectionavailable after all assets are received and verified is logically thesame for any collection of digital assets no matter their type, orsource.

This type of digital asset quarantine and verification procedure usingcontent-addressable asset identifiers ensures asset integrity, excludesspoofing and virus infection automatically, and permits automated ormanual reconstruction of lost assets. The content-addressable assetidentifier system of the present invention is superior to other assetidentification systems where identifiers are not derived from thedigital asset contents but instead depend on a path name locator, filename, file author, file creation/modification date, file size, or otherenvironmental or application meta-data. Because such prior artidentifiers are not content addressable, they may be readily spoofed. Bycontrast, the verify step 404 of the present invention allows errors inassets to be detected. Advantageously, under the present invention, iferrors are detected and a virus or spoofing is suspected, suspect assetsmay be reconstituted from another location or a more secure method offile transfer may be invoked.

FIG. 5 is a flowchart illustrating one embodiment of step 402 showinghow the importer checks “multicast” transmissions in order toreconstitute assets that are received in portions. Multicasttransmissions are transmissions from a peer that are addressed to allpeers available on the network. Similarly, a multicast request may besent by sending a request to all peers available on the network. Peersinclude any device included in a defined multicast group; a multicastgroup may include any device accessible over a data link. This method isreferred to as “Swiss cheese” method because it fills in assets byplacing the chunks of an asset in the proper order and continues torequest chunks that are needed to fill in the holes. The method permitsmultiple source, variable, nonsequential digital asset segment transferin response to a request using a content-addressable asset name (such asa cryptographic hash asset identifier). Of course, other methods forreceiving files may also be used.

In step 510 the importer receives a multicast transmission. Next, instep 520 the importer checks its asset request list (akin to a shoppinglist) to see if a digital asset portion or segment received is needed.If the asset segment is not needed, control is transferred back to step510. If the asset segment is needed, then the data is stored in theproper order based on its sequence number in step 530 and the assetrequest list is updated so that that particular asset segment will notbe requested any longer. The process then ends.

It should be noted in the above described protocol that digital assets(binary sequences) are received in parts as portions or segments andthat the asset request list includes all of the assets that are beingrequested until those assets (binary sequences) are received in theirentirety. In other embodiments, assets may be received whole or in amanner specified by any file transfer protocol. It is also possible thatin some embodiments, an asset segment request list would be implementedthat would include individual segments being requested. For example,individual segments of assets may be requested when data or a code patchfor a software application is required, or when specific entries for adatabase are obtained by a store or query result. The importer managesthe transfer of assets to the recipient of an e-CLIP and determines whenthe assets are complete so that the cryptographic hash file identifierspecified in the descriptor file may be used to verify that the correctasset has been received. Verification is achieved by comparing thecryptographic hash asset identifier to a newly generated MD5cryptographic hash asset identifier calculated using the received asset(binary sequence).

FIG. 6A is a block diagram illustrating the structure of an assetrequest generated by an importer as described above in step 402. Arequest 600 includes an asset identifier 602, a sequence number 604, anda chunk size 606. The asset identifier is obtained from the descriptorfile. The sequence numbers may be generated by the importer based on thesize of the asset segments that it will request. The chunk size isspecified by the importer in certain embodiments. It should be notedthat in other embodiments, the chunk size is specified by the system andis not changeable by individual file importers.

FIG. 6B is a block diagram illustrating the structure of a data packetthat delivers binary asset data to a requester in response to an assetrequest. A data packet 610 includes an asset identifier 612, a sequencenumber 614, and data 616 which represents the asset segment data itself.The length of the data corresponds to the length of the chunk size 606specified in the request 600. Thus, incoming asset segments can beordered according to their sequence number and the data in the file canbe recovered from the ordered segments once all of the segments havebeen received. Notably, portions may be received from different sourcesin nonsequential order and concatenated or filled in to create thetarget digital asset (binary sequence).

In one embodiment, the importer has a specific hierarchy of locations ina computer system (or on a network) in which it looks for the assetslisted in the descriptor file. Thus, the importer may be implementedusing a chained system of importers which look for digital assets indifferent places.

FIG. 7 is a block diagram illustrating one such chained set ofimporters. A verifier importer 702 attempts to first verify that thedigital asset is stored on a local disk in asset storage. Asset storageis an area of local memory reserved for storing data in a binary form ina way optimized for instant retrieval using a cryptographic hash fileidentifier. If the verifier importer finds the digital asset in assetstorage, then the verifier checks the cryptographic hash assetidentifier by calculating it and then verifies that the asset in assetstorage is actually the asset being requested.

If the verifier importer is not able to find the digital asset in assetstorage, then a find importer 704 is enabled to locate the asset (if afile) in local conventional storage, if possible. If the asset is a fileand is not found in local conventional storage or is some other form ofdigital asset, then a multicast importer 706 such as the one describedabove in FIG. 5 is enabled to broadcast signals within the multi-castgroup of the recipient of the e-CLIP to attempt to obtain the assetsspecified by the e-CLIP. If the assets can not be obtained by themulti-cast importer, then a copy importer 708 is used to look for theasset stored as a content-addressable file on any mounted volumes onfile servers that are accessible to the copy importer. If the assetsstill are not found, then a download importer 710 is used to downloadthe asset stored as a content-addressable file from an FTP server orsome other outside source. As noted above, certain e-CLIPs may include aresource locator such as a URL that specifies a specific outsidelocation where the assets stored as files included in the e-CLIP may befound if they are not obtainable by any of the other importers aboveimporter 710. Alternatively, a traditional file transfer request can beused.

Thus, the importers are in a hierarchy and assets are searched for firstin the most convenient location and then in progressively lessconvenient locations. This “assembly line” of importers is configurablein kind and quantity of importers and may automatically and dynamicallychange to optimize economy, security or performance. Because thecryptographic hash asset identifiers serve as content-based file namesthat enable the content of assets to be verified once the assets arerecovered, it is possible to allow assets to be recovered from arbitrarylocations where they may be found without regard to checking thecontents of the asset using some sort of check sum. Advantageously, thecryptographic hash asset identifier acts as both a digital asset (binarysequence) identifier and a means for verifying the asset contents.

CONCLUSION

In one embodiment, a system and method has been described for specifyinga collection of files having an arbitrary directory structure to bereconstructed from whatever sources are available to the target system.The files are described in a directory structure in a descriptor fileand a cryptographic hash file list identifier (e-CLIP) is generated forthe descriptor file. The e-CLIP represents the collection of files andmay then be transferred, stored, etc.

When the descriptor file is obtained by the recipient using the e-CLIP,the descriptor file is verified as the correct descriptor file specifiedby the e-CLIP using the same algorithm that was used to generate thecryptographic hash file list identifier. Then, each of the filesspecified in the descriptor file are recovered using an importer and thefiles are verified using the cryptographic hash file identifiers foreach file. Thus, information is obtainable by a recipient using meansmore efficient than simply receiving all of the information over onecommunication line from the information sender's location. Theinformation may be reliably gathered by the recipient of the e-CLIPbecause the e-CLIP contains cryptographic hash file identifiers for eachfile that are used to verify the contents of the files.

In addition to specifying files for transfer from one entity to another,the e-CLIP described herein can also be used to create a record of theexact state of any collection of files in a computer at any given time.This is done by generating an e-CLIP that is a cryptographic hash fileidentifier of a descriptor file that includes directory information forthat collection of files in the computer. Preferably, all of the filesare first backed up elsewhere for later retrieval if necessary. If thecomputer files are lost for any reason, the e-CLIP is used to retrievethe descriptor file (which has been stored in a safe location). Thedescriptor file can then be used to retrieve all of the files that arereferenced within it, either a locally or over a network. Preferably,the importers described herein are used to retrieve the files.

Thus, the state of the files in the computer may be recorded exactly bysimply generating an e-CLIP for the files, storing the e-CLIP safely,and making sure copies of the files exist elsewhere. The files may berecovered if needed by retrieving the e-CLIP, using the e-CLIP to findthe descriptor file, opening the descriptor file, and then using theimporters to retrieve the correct versions of all of the filesrepresented therein.

This is an efficient way to back up multiple computers on a network whenmany of the computers contain the same files. Each computer on thenetwork generates a descriptor file describing all of its files as wellas its directory structure. The descriptor file is sent to a centralbackup computer that makes certain that it contains all of the filesspecified in each of the descriptor files. The amount of datacompression achieved by this scheme can be extremely large when thecomputers being backed up contain many common files as is the case withPersonal Computers on Local Area Networks (LANs). Each file need only beobtained and stored once by the central backup computer and thenspecified as many times as a necessary in the individual descriptorfiles that represent the files found on the individual computers.

Subsequent backups of the same computers may be accomplished bygenerating a new descriptor file which includes file hashes for the newor changed files, and having the central computer store the newdescriptor file and all of the newly created or changed files. Thus, newor changed files may be reliably identified and copied to the centralbackup computer without moving previously archived, unmodified files.Storing periodic backups for each computer can thus be accomplishedwithout requiring prohibitive amounts of file storage since each newbackup only requires additional storage for new or changed files.

Similar mechanisms will be embodied which perform archiving ofindividual records of databases, Web pages, and/or any other digitalassets which may be identified by some process and made available tomechanisms identical in nature to those described above.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications may be practiced within the scope of theappended claims. For example, a wide variety of algorithms may be usedto compute a unique file identifier for an asset, and for the descriptorfile. Also, the asset list hash may be represented in any suitable formsuch as binary, another suitable base, ASCII, alphanumeric, etc. Thee-CLIP, descriptor file and individual assets may be stored in the samelocation or in different locations. Many different forms may be used forthe descriptor file; it may contain solely the asset hash for eachdigital asset (binary sequence), or a wide variety of other information.Assets may be retrieved using the importers described herein, or usingother techniques. Assets may be verified only if there is an exactmatch, or some room for error to allow for minor changes in files mayalso be acceptable. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the appended claims.

Computer System Embodiment

FIGS. 8 and 9 illustrate a computer system 900 suitable for implementingembodiments of the present invention. FIG. 8 shows one possible physicalform of the computer system. Of course, the computer system may havemany physical forms ranging from an integrated circuit, a printedcircuit board and a small handheld device up to a huge super computer.Computer system 900 includes a monitor 902, a display 904, a housing906, a disk drive 908, a keyboard 910 and a mouse 912. Disk 914 is acomputer-readable medium used to transfer data to and from computersystem 900.

FIG. 9 is an example of a block diagram for computer system 900.Attached to system bus 920 are a wide variety of subsystems.Processor(s) 922 (also referred to as central processing units, or CPUs)are coupled to storage devices including memory 924. Memory 924 includesrandom access memory (RAM) and read-only memory (ROM). As is well knownin the art, ROM acts to transfer data and instructions uni-directionallyto the CPU and RAM is used typically to transfer data and instructionsin a bi-directional manner. Both of these types of memories may includeany suitable of the computer-readable media described below. A fixeddisk 926 is also coupled bi-directionally to CPU 922; it providesadditional data storage capacity and may also include any of thecomputer-readable media described below. Fixed disk 926 may be used tostore programs, data and the like and is typically a secondary storagemedium (such as a hard disk) that is slower than primary storage. Itwill be appreciated that the information retained within fixed disk 926,may, in appropriate cases, be incorporated in standard fashion asvirtual memory in memory 924. Removable disk 914 may take the form ofany of the computer-readable media described below.

CPU 922 is also coupled to a variety of input/output devices such asdisplay 904, keyboard 910, mouse 912 and speakers 930. In general, aninput/output device may be any of: video displays, track balls, mice,keyboards, microphones, touch-sensitive displays, transducer cardreaders, magnetic or paper tape readers, tablets, styluses, voice orhandwriting recognizers, biometrics readers, or other computers. CPU 922optionally may be coupled to another computer or telecommunicationsnetwork using network interface 940. With such a network interface, itis contemplated that the CPU might receive information from the network,or might output information to the network in the course of performingthe above-described method steps. Furthermore, method embodiments of thepresent invention may execute solely upon CPU 922 or may execute over anetwork such as the Internet in conjunction with a remote CPU thatshares a portion of the processing.

In addition, embodiments of the present invention further relate tocomputer storage products with a computer-readable medium that havecomputer code thereon for performing various computer-implementedoperations. The media and computer code may be those specially designedand constructed for the purposes of the present invention, or they maybe of the kind well known and available to those having skill in thecomputer software arts. Examples of computer-readable media include, butare not limited to: magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROMs and holographic devicesmagneto-optical media such as floptical disks; and hardware devices thatare specially configured to store and execute program code, such asapplication-specific integrated circuits (ASICs), programmable logicdevices (PLDs) and ROM and RAM devices. Examples of computer codeinclude machine code, such as produced by a compiler, and filescontaining higher level code that are executed by a computer using aninterpreter.

1. A method for use in a computer system comprising at least first andsecond computers, wherein the first computer provides contentaddressable storage, the method comprising acts of: (A) receiving, atthe first computer, a request to access a unit of data stored by thefirst computer, the request identifying the unit of data via anidentifier that is based, at least in part, on the content of thedigital asset; (B) using the identifier to locate the unit of data. 2.The method of claim 1, wherein the request identifies the secondcomputer as having issued the request, and wherein the method furthercomprises an act of: (C) returning the unit of data to the secondcomputer.
 3. The method of claim 1, wherein the unit of data comprisesat least one digital asset
 4. The method of claim 1, wherein the unit ofdata comprises at least one descriptor file.
 5. The method of claim 1,wherein the digital asset comprises at least one unit of data andmetadata relating to at least one unit of data.
 6. The method of claim1, wherein the identifier is generated by applying a cryptographic hashfunction to at least a portion of the content of the unit of data. 7.The method of claim 6, wherein the identifier is generated by applying acryptographic hash function to the entire content of the unit of data.8. The method of claim 4, wherein the descriptor file includes a secondidentifier for at least one digital asset and metadata relating to theat least one digital asset, wherein the second identifier is based uponthe content of the at least one digital asset.
 9. The method of claim 8,wherein the request is a request to delete the descriptor file, andwherein the method further comprises an act of: (C) deleting thedescriptor file in response to receipt of the request.
 10. The method ofclaim 1, wherein the unit of data is a digital asset, wherein therequest is a request to delete the digital asset, and wherein the methodfurther comprises an act of: (C) deleting the digital asset in responseto receipt of the request.
 11. The method of claim 1, wherein theidentifier in the request is a first identifier for the unit of data,and wherein the method further comprises acts of: (C)— generating, atthe first computer, a second identifier for the unit of data based uponthe content of the unit of data stored by the first computer; and (D)comparing the second identifier to the first identifier to verify thatthe first and second identifiers are identical.
 12. The method of claim11, wherein the first identifier is generated by applying acryptographic hash function to at least a portion of the content of theunit of data and wherein the act (C) comprises an act of generating thesecond identifier using the cryptographic hash function on the at leasta portion of the content of the unit of data stored by the firstcomputer.
 13. The method of claim 11, wherein the first identifier isgenerated by applying a cryptographic hash function to the entirecontent of the unit of data, and wherein the act (C) comprises an act ofgenerating the second identifier using the cryptographic hash functionon the entire content of the unit of data stored by the first computer.14. The method of claim 1, wherein the first computer comprises a filesystem to manage units of data stored by the first computer, wherein theidentifier is used as the name of a file in which the unit of data isstored by the storage system, and wherein the act (B) comprises an actof using the identifier to index into the file system to locate the unitof data.
 15. The method of claim 14, wherein the identifier is generatedby applying a cryptographic hash function to at least a portion of thecontent of the unit of data.
 16. The method of claim 15, wherein theidentifier is generated by applying a cryptographic hash function to theentire content of the unit of data.
 17. The method of claim 1, whereinthe first computer is a silo.
 18. The method of claim 1, wherein thefirst computer is a file server.
 19. The method of claim 1, wherein thefirst computer is a storage system.
 20. The method of claim 1, whereinthe computer system comprises a plurality of computers in addition tothe first and second computers and wherein the plurality of computersare interconnected by at least one network, and wherein each of theplurality of computers manages storage of one or more units of data. 21.The method of claim 1, wherein the first and second computers are peercomputers coupled by at least one network.
 22. The method of claim 1,wherein the unit of data is stored on the first computer.
 23. The methodof claim 1, wherein the unit of data is stored by the first computer ona storage device managed by the first computer.
 24. At least onecomputer readable medium encoded with instructions that, when executedon a computer system, perform a method, the computer system comprisingat least first and second computers, wherein the first computer providescontent addressable storage, and wherein the method comprises acts of:(A) receiving a request to access a unit of data stored by the firstcomputer, the request identifying the unit of data via an identifierthat is based, at least in part, on the content of the digital asset;(B) using the identifier to locate the unit of data.
 25. The at leastone computer readable medium of claim 24, wherein the request identifiesthe second computer as having issued the request, and wherein the methodfurther comprises an act of: (C) returning the unit of data to thesecond computer.
 26. The at least one computer readable medium of claim24, wherein the unit of data comprises at least one digital asset 27.The at least one computer readable medium of claim 24, wherein the unitof data comprises at least one descriptor file.
 28. The at least onecomputer readable medium of claim 24, wherein the digital assetcomprises at least one unit of data and metadata relating to at leastone unit of data.
 29. The at least one computer readable medium of claim24, wherein the identifier is generated by applying a cryptographic hashfunction to at least a portion of the content of the unit of data. 30.The at least one computer readable medium of claim 29, wherein theidentifier is generated by applying a cryptographic hash function to theentire content of the unit of data.
 31. The at least one computerreadable medium of claim 27, wherein the descriptor file includes asecond identifier for at least one digital asset and metadata relatingto the at least one digital asset, wherein the second identifier isbased upon the content of the at least one digital asset.
 32. The atleast one computer readable medium of claim 31, wherein the request is arequest to delete the descriptor file, and wherein the method furthercomprises an act of: (C) deleting the descriptor file in response toreceipt of the request. 33 The at least one computer readable medium ofclaim 24, wherein the unit of data is a digital asset, wherein therequest is a request to delete the digital asset, and wherein the methodfurther comprises an act of: (C) deleting the digital asset in responseto receipt of the request.
 34. The at least one computer readable mediumof claim 24, wherein the identifier in the request is a first identifierfor the unit of data, and wherein the method further comprises acts of:(C) generating, at the first computer, a second identifier for the unitof data based upon the content of the unit of data stored by the firstcomputer; and (D) comparing the second identifier to the firstidentifier to verify that the first and second identifiers areidentical.
 35. The at least one computer readable medium of claim 34,wherein the first identifier is generated by applying a cryptographichash function to at least a portion of the content of the unit of dataand wherein the act (C) comprises an act of generating the secondidentifier using the cryptographic hash function on the at least aportion of the content of the unit of data stored by the first computer.36. The at least one computer readable medium of claim 34, wherein thefirst identifier is generated by applying a cryptographic hash functionto the entire content of the unit of data, and wherein the act (C)comprises an act of generating the second identifier using thecryptographic hash function on the entire content of the unit of datastored by the first computer.
 37. The at least one computer readablemedium of claim 24, wherein the first computer comprises a file systemto manage units of data stored by the first computer, wherein theidentifier is used as the name of a file in which the unit of data isstored by the storage system, and wherein the act (B) comprises an actof using the identifier to index into the file system to locate the unitof data.
 38. The at least one computer readable medium of claim 37,wherein the identifier is generated by applying a cryptographic hashfunction to at least a portion of the content of the unit of data. 39.The at least one computer readable medium of claim 38, wherein theidentifier is generated by applying a cryptographic hash function to theentire content of the unit of data.
 40. The at least one computerreadable medium of claim 24, wherein the first computer is a silo. 41.The at least one computer readable medium of claim 24, wherein the firstcomputer is a file server.
 42. The at least one computer readable mediumof claim 24, wherein the first computer is a storage system.
 43. The atleast one computer readable medium of claim 24, wherein the computersystem comprises a plurality of computers in addition to the first andsecond computers and wherein the plurality of computers areinterconnected by at least one network, and wherein each of theplurality of computers manages storage of one or more units of data. 44.The at least one computer readable medium of claim 24, wherein the firstand second computers are peer computers coupled by at least one network.45. The at least one computer readable medium of claim 24, wherein theunit of data is stored on the first computer.
 46. The at least onecomputer readable medium of claim 24, wherein the unit of data is storedby the first computer on a storage device managed by the first computer.47. A first computer that provides content addressable storage for usein a computer system comprising the first computer and a secondcomputer, the first computer comprising: an input for receiving accessrequests; and at least one controller, coupled to the input, that:receives, through the input, a request to access a unit of data storedby the first computer, the request identifying the unit of data via anidentifier that is based, at least in part, on the content of thedigital asset; and uses the identifier to locate the unit of data. 48.The first computer of claim 47, wherein the request identifies thesecond computer as having issued the request, and wherein the at leastone controller returns the unit of data to the second computer.
 49. Thefirst computer of claim 47, wherein the unit of data comprises at leastone digital asset
 50. The first computer of claim 47, wherein the unitof data comprises at least one descriptor file.
 51. The first computerof claim 47, wherein the digital asset comprises at least one unit ofdata and metadata relating to at least one unit of data.
 52. The firstcomputer of claim 47, wherein the identifier is generated by applying acryptographic hash function to at least a portion of the content of theunit of data.
 53. The first computer of claim 52, wherein the identifieris generated by applying a cryptographic hash function to the entirecontent of the unit of data.
 54. The first computer of claim 50, whereinthe descriptor file includes a second identifier for at least onedigital asset and metadata relating to the at least one digital asset,wherein the second identifier is based upon the content of the at leastone digital asset.
 55. The first computer of claim 54, wherein therequest is a request to delete the descriptor file, and wherein the atleast one controller deletes the descriptor file in response to receiptof the request.
 56. The first computer of claim 47, wherein the unit ofdata is a digital asset, wherein the request is a request to delete thedigital asset, and wherein the at least one controller deletes thedigital asset in response to receipt of the request.
 57. The firstcomputer of claim 47, wherein the identifier in the request is a firstidentifier for the unit of data, and wherein the at least onecontroller: generates a second identifier for the unit of data basedupon the content of the unit of data stored by the first computer; andcompares the second identifier to the first identifier to verify thatthe first and second identifiers are identical.
 58. The first computerof claim 57, wherein the first identifier is generated by applying acryptographic hash function to at least a portion of the content of theunit of data and wherein the at least one controller generates thesecond identifier using the cryptographic hash function on the at leasta portion of the content of the unit of data stored by the firstcomputer.
 59. The first computer of claim 57, wherein the firstidentifier is generated by applying a cryptographic hash function to theentire content of the unit of data, and wherein the at least onecontroller generates the second identifier using the cryptographic hashfunction on the entire content of the unit of data stored by the firstcomputer.
 60. The first computer of claim 47, wherein the first computercomprises a file system to manage units of data stored by the firstcomputer, wherein the identifier is used as the name of a file in whichthe unit of data is stored by the storage system, and wherein the atleast one controller uses the identifier to index into the file systemto locate the unit of data.
 61. The first computer of claim 60, whereinthe identifier is generated by applying a cryptographic hash function toat least a portion of the content of the unit of data.
 62. The firstcomputer of claim 61, wherein the identifier is generated by applying acryptographic hash function to the entire content of the unit of data.63. The first computer of claim 47, wherein the first computer is asilo.
 64. The first computer of claim 47, wherein the first computer isa file server.
 65. The first computer of claim 47, wherein the firstcomputer is a storage system.
 66. The first computer of claim 47,wherein the computer system comprises a plurality of computers inaddition to the first and second computers and wherein the plurality ofcomputers are interconnected by at least one network, and wherein eachof the plurality of computers manages storage of one or more units ofdata.
 67. The first computer of claim 47, wherein the first and secondcomputers are peer computers coupled by at least one network.
 68. Thefirst computer of claim 47, wherein the unit of data is stored on thefirst computer.
 69. The first computer of claim 47, wherein the unit ofdata is stored by the first computer on a storage device managed by thefirst computer. Host Read
 70. A method for use in a computer systemcomprising at least first and second computers, wherein the firstcomputer provides content addressable storage, the method comprising anact of: (A) transmitting, from the second computer to the firstcomputer, a request to access a unit of data stored by the firstcomputer, the request identifying the digital asset via an identifierthat is based, at least in part, on the content of the unit of data. 71.The method of claim 70, wherein the unit of data comprises at least onedigital asset.
 72. The method of claim 70, wherein the unit of datacomprises at least one descriptor file.
 73. The method of claim 70,wherein the unit of data comprises at least one digital asset and atleast one descriptor file.
 74. The method of claim 70, wherein the actof transmitting comprises an act of transmitting a request thatidentifies the unit of data via an identifier that was generated byapplying a cryptographic hash function to at least a portion of thecontent of the unit of data.
 75. The method of claim 74, wherein the actof transmitting comprises an act of transmitting a request thatidentifies the unit of data via an identifier that was generated byapplying a cryptographic hash function to the entire content of the unitof data.
 76. The method of claim 72, wherein the descriptor fileincludes a second identifier for at least one digital asset and metadatarelating to the at least one digital asset, wherein the secondidentifier is based, at least in part, on the content of the at leastone digital asset.
 77. The method of claim 76, wherein the act oftransmitting the request comprises transmitting a request to delete thedescriptor file.
 78. The method of claim 81, wherein the act oftransmitting the request comprises transmitting a request to delete thedigital asset.
 79. The method of claim 70, further comprising an act of:(B) receiving the unit of data at the second computer.
 80. The method ofclaim 79, wherein the identifier in the request is a first identifierfor the digital asset, and wherein the method further comprises acts of:(C) receiving, at the second computer, a second identifier for thedigital asset; and (D) comparing the second identifier to the firstidentifier to verify that the first and second identifiers areidentical.
 81. The method of claim 80, wherein the first identifier wasgenerated by applying a cryptographic hash function to at least aportion of the content of the unit of data, and wherein the act ofreceiving a second identifier comprises an act of receiving a secondidentifier that was generated by applying the cryptographic hashfunction to the at least a portion of the content of the unit of datastored by the first computer.
 82. The method of claim 80, wherein thefirst identifier was generated by applying a cryptographic hash functionto the entire content of the unit of data, and wherein the act ofreceiving a second identifier comprises an act of receiving a secondidentifier that was generated by applying the cryptographic hashfunction to the entire content of the unit of data stored by the firstcomputer.
 83. The method of claim 70, wherein the first computer is asilo.
 84. The method of claim 70, wherein the first computer is a fileserver.
 85. The method of claim 70, wherein the first computer is astorage system.
 86. The method of claim 70, wherein the computer systemcomprises a plurality of computers in addition to the first and secondcomputers, wherein the plurality of computers are interconnected by atleast one network, and wherein each of the plurality of computersmanages storage of one or more digital assets.
 87. The method of claim70, wherein the first and second computers are peer computers coupled byat least one network.
 88. The method of claim 79, wherein the identifierin the request is a first identifier for the unit of data, and whereinthe method further comprises acts of: (C) generating, at the secondcomputer, a second identifier for the unit of data received at thesecond computer; and (D) comparing the second identifier to the firstidentifier to verify that the first and second identifiers areidentical.
 89. The method of claim 88, wherein the first identifier wasgenerated by applying a cryptographic hash function to at least aportion of the content of the unit of data, and wherein the act (C)comprises an act of generating the second identifier by applying thecryptographic hash function to the at least a portion of the content ofthe unit of data received at the second computer.
 90. The method ofclaim 88, wherein the first identifier was generated by applying acryptographic hash function to the entire content of the unit of data,and wherein the act (C) comprises an act of generating the secondidentifier by applying the cryptographic hash function to the entirecontent of the unit of data received by the second computer.
 91. Atleast one computer readable medium encoded with instructions that, whenexecuted on a computer system, perform a method, wherein the computersystem comprises at least first and second computers, wherein the firstcomputer provides content addressable storage, and wherein the methodcomprises an act of: (A) transmitting, from the second computer to thefirst computer, a request to access a unit of data stored by the firstcomputer, the request identifying the digital asset via an identifierthat is based, at least in part, on the content of the unit of data. 92.The at least one computer readable medium of claim 91, wherein the unitof data comprises at least one digital asset.
 93. The at least onecomputer readable medium of claim 91, wherein the unit of data comprisesat least one descriptor file.
 94. The at least one computer readablemedium of claim 91, wherein the unit of data comprises at least onedigital asset and at least one descriptor file.
 95. The at least onecomputer readable medium of claim 91, wherein the act of transmittingcomprises an act of transmitting a request that identifies the unit ofdata via an identifier that was generated by applying a cryptographichash function to at least a portion of the content of the unit of data.96. The at least one computer readable medium of claim 95, wherein theact of transmitting comprises an act of transmitting a request thatidentifies the unit of data via an identifier that was generated byapplying a cryptographic hash function to the entire content of the unitof data.
 97. The at least one computer readable medium of claim 93,wherein the descriptor file includes a second identifier for at leastone digital asset and metadata relating to the at least one digitalasset, wherein the second identifier is based, at least in part, on thecontent of the at least one digital asset.
 98. The at least one computerreadable medium of claim 97, wherein the act of transmitting the requestcomprises transmitting a request to delete the descriptor file.
 99. Theat least one computer readable medium of claim 92, wherein the act oftransmitting the request comprises transmitting a request to delete thedigital asset.
 100. The at least one computer readable medium of claim91, wherein the method further comprises an act of: (B) receiving theunit of data at the second computer.
 101. The at least one computerreadable medium of claim 100, wherein the identifier in the request is afirst identifier for the digital asset, and wherein the method furthercomprises acts of: (C) receiving, at the second computer, a secondidentifier for the digital asset; and (D) comparing the secondidentifier to the first identifier to verify that the first and secondidentifiers are identical.
 102. The at least one computer readablemedium of claim 101, wherein the first identifier was generated byapplying a cryptographic hash function to at least a portion of thecontent of the unit of data, and wherein the act of receiving a secondidentifier comprises an act of receiving a second identifier that-wasgenerated by applying the cryptographic hash function to the at least aportion of the content of the unit of data stored by the first computer.103. The at least one computer readable medium of claim 101, wherein thefirst identifier was generated by applying a cryptographic hash functionto the entire content of the unit of data, and wherein the act ofreceiving a second identifier comprises an act of receiving a secondidentifier that was generated by applying the cryptographic hashfunction to the entire content of the unit of data stored by the firstcomputer.
 104. The at least one computer readable medium of claim 91,wherein the first computer is a silo.
 105. The at least one computerreadable medium of claim 91, wherein the first computer is a fileserver.
 106. The at least one computer readable medium of claim 91,wherein the first computer is a storage system.
 107. The at least onecomputer readable medium of claim 91, wherein the computer systemcomprises a plurality of computers in addition to the first and secondcomputers, wherein the plurality of computers are interconnected by atleast one network, and wherein each of the plurality of computersmanages storage of one or more digital assets.
 108. The at least onecomputer readable medium of claim 91, wherein the first and secondcomputers are peer computers coupled by at least one network.
 109. Theat least one computer readable medium of claim 100, wherein theidentifier in the request is a first identifier for the unit of data,and wherein the method further comprises acts of: (C) generating, at thesecond computer, a second identifier for the unit of data received atthe second computer; and (D) comparing the second identifier to thefirst identifier to verify-that the first and second identifiers areidentical.
 110. The at least one computer readable medium of claim 109,wherein the first identifier was generated by applying a cryptographichash function to at least a portion of the content of the unit of data,and wherein the act (C) comprises an act of generating the secondidentifier by applying the cryptographic hash function to the at least aportion of the content of the unit of data received at the secondcomputer.
 111. The at least one computer readable medium of claim 109,wherein the first identifier was generated by applying a cryptographichash function to the entire content of the unit of data, and wherein theact (C) comprises an act of generating the second identifier by applyingthe cryptographic hash function to the entire content of the unit ofdata received by the second computer.
 112. A first computer thatprovides content addressable storage for use in a computer systemcomprising the first computer and a second computer, the first computercomprising: an output for transmitting access requests; at least onecontroller, coupled to the output, that: transmits, to the secondcomputer, a request to access a unit of data stored by the secondcomputer, the request identifying the digital asset via an identifierthat is based, at least in part, on the content of the unit of data.113. The first computer of claim 112, wherein the unit of data comprisesat least one digital asset.
 114. The first computer of claim 112,wherein the unit of data comprises at least one descriptor file. 115.The first computer of claim 112, wherein the unit of data comprises atleast one digital asset and at least one descriptor file.
 116. The firstcomputer of claim 112, wherein the at least one controller transmits arequest that identifies the unit of data via an identifier that wasgenerated by applying a cryptographic hash function to at least aportion of the content of the unit of data.
 117. The first computer ofclaim 116, wherein at least one controller transmits a request thatidentifies the unit of data via an identifier that was generated byapplying a cryptographic hash function to the entire content of the unitof data.
 118. The first computer of claim 114, wherein the descriptorfile includes a second identifier for at least one digital asset andmetadata relating to the at least one digital asset, wherein the secondidentifier is based, at least in part, on the content of the at leastone digital asset.
 119. The first computer of claim 118, wherein the atleast one controller transmits a request to delete the descriptor file.120. The first computer of claim 113, wherein the at least onecontroller transmits a request to delete the digital asset.
 121. Thefirst computer of claim 112, wherein the at least one controllerreceives the unit of data.
 122. The first computer of claim 121, whereinthe identifier in the request is a first identifier for the digitalasset, and wherein the at least one controller: receives a secondidentifier for the digital asset; and compares the second identifier tothe first identifier to verify that the first and second identifiers areidentical.
 123. The first computer of claim 122, wherein the firstidentifier was generated by applying a cryptographic hash function to atleast a portion of the content of the unit of data, and wherein the atleast one controller receives a second identifier that was generated byapplying the cryptographic hash function to the at least a portion ofthe content of the unit of data stored by the second computer.
 124. Thefirst computer of claim 122, wherein the first identifier was generatedby applying a cryptographic hash function to the entire content of theunit of data, and wherein at least one controller receives a secondidentifier that was generated by applying the cryptographic hashfunction to the entire content of the unit of data stored by the secondcomputer.
 125. The first computer of claim 112, wherein the secondcomputer is a silo.
 126. The first computer of claim 112, wherein thesecond computer is a file server.
 127. The first computer of claim 112,wherein the second computer is a storage system.
 128. The first computerof claim 112, wherein the computer system comprises a plurality ofcomputers in addition to the first and first computers, wherein theplurality of computers are interconnected by at least one network, andwherein each of the plurality of computers manages storage of one ormore digital assets.
 129. The first computer of claim 112, wherein thefirst and second computers are peer computers coupled by at least onenetwork.
 130. The first computer of claim 121, wherein the identifier inthe request is a first identifier for the unit of data, and wherein theat least one controller: generates a second identifier for the unit ofdata received at the first computer; and compares the second identifierto the first identifier to verify that the first and second identifiersare identical.
 131. The first computer of claim 130, wherein the firstidentifier was generated by applying a cryptographic hash function to atleast a portion of the content of the unit of data, and wherein the atleast one controller generates the second identifier by applying thecryptographic hash function to the at least a portion of the content ofthe unit of data.
 132. The first computer of claim 112, wherein thesecond computer is a file server.
 133. The method of claim 1, whereinthe request identifies the second computer as having issued the request,and wherein the method further comprises an act of: returning, to thesecond computer, information indicating whether the unit of data isstored on the first computer.
 134. The at least one computer readablemedium of claim 24, wherein the request identifies the second computeras having issued the request, and wherein the method further comprisesan act of: returning, to the second computer, information indicatingwhether the unit of data is stored on the first computer.
 135. The firstcomputer of claim 47, wherein the request identifies the second computeras having issued the request, and wherein the at least one controllerreturns, to the second computer, information indicating whether the unitof data is stored on the first computer.